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Crew Decision Making Trends and Issues 


1. INTRODUCTION 


1.1 Purpose of Study 

The goal of the Vehicle System Safety Technology (VSST) project within NASA’s Aviation 
Safety Program (AvSP) is to “enable a reduction in accidents and incidents through enhanced 
vehicle design, structure, systems, and operating concepts” (ref. 1). The VSST project is 
composed of three key Technical Challenges (TCs): 

(1) Improve Crew Decision-Making and Response in Complex Situations (CDM) 

(2) Maintain Vehicle Safety between Major Inspections (MVS) 

(3) Assure Safe and Effective Aircraft Control under Hazardous Conditions (ASC) 

VSST project management uses systems analysis “to identify key issues and maintain a 
portfolio of research leading to potential solutions to these challenges”. To assist management 
in accomplishing these objectives, several systems analysis related milestones have been 
specified in the VSST project plan. The systems analysis milestone for fiscal year (FY) 2013 is 
focused on the CDM Technical Challenge. The goal of CDM is to “develop and demonstrate 
new capabilities that enable pilots to better understand and respond to complex situations”. The 
expected research outcome of the CDM TC is the set of five research products listed in Table 1. 
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Table 1. CDM List of Research Products 


IMPROVE CREW DECISION-MAKING AND RESPONSE IN COMPLEX SITUATIONS (CDM) 

CDM-1 

Advanced Displays for 
Terminal Area and 
Surface Operations 

Advanced displays and decision support concepts that improve 
situation awareness and provide hazard protection during 
terminal area and runway /surface current-day and NextGen 
operations 

CDM-2 

Flight Deck 
Countermeasures for 
Spatial Disorientation 
and Loss-of-Energy State 
Awareness 

Displays, decision support tools, and Crew Resource 
Management (CRM) concepts to help crews avoid, detect, 
mitigate, and recover from the loss-of-aircraft attitude and 
energy state awareness 

CDM-3 

Crew System Monitoring 
And Management System 
(CSMMS) 

These technologies offer the capability of identifying and 
quantifying the current physiological, workload, and attentional 
state of the crew and these data may be used to ensure and 
adapt to the crew’s and/or automation system’s readiness to 
respond in potentially hazardous situations. 

CDM-4 

Flight Deck Information 
Management Systems for 
Integrity and Awareness 

The primary objective of this technology is the creation of 
integrated flight deck information management systems that 
assure the information needed by flight crews to make critical 
decisions is complete and not misleading. 

CDM-5 

Revised Pilot Proficiency 
Standards for Manual 
Handling and Automation 
Interactions 

This technology will result in the creation of data and guidelines 
to support revised pilot proficiency standards and training 
associated with manual handling, automation interactions, and 
reverting from automated to manual handling. 


As detailed in the VSST Project Plan, the CDM TC is described as follows: 

“ CDM research develops flight deck capabilities that enable pilots to make more 
informed decisions when confronted with complex situations. From a review of 
recent accidents, pilots are increasingly faced with complex, multi-faceted 
situations. Prior emphasis on responding to an individual failure with a targeted 
checklist may not always apply. Given the complexities of current-day operations 
and future trends toward greater levels of automation and information, 
technologies must be developed that help pilots assess these situations and 
execute an informed course of action. CDM research focuses on low altitude 
terminal area operations since this phase of flight is at greatest risk due to its 
operational complexity and time criticality. “ 


A summary of the goals, objectives and products for the CDM TC are graphically depicted using 
an objectives tree format (ref. 2) in Figure 1. 
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Figure 1. CDM Objectives Tree 


The specific CDM related FY13 systems analysis milestone is stated below: 

(1) “Deliver analysis of trends in aviation as related to crew systems interactions & decisions 
by reviewing the most current statistical and prognostic data available about accidents 
and incidents related to crew systems research areas 

(2) Deliver assessment of future directions in aviation technology related to crew systems 
research areas through review of literature from academia, industry and other 
government agencies to establish requirements for future work in crew systems” 


1 .2 Overview of Study Contents 

The expected outcomes for this study are addressed in sequential order. Outcome 1 is 
addressed in section 2, which contains a summary of statistical analyses of accident and 
incident data that have been conducted by NASA researchers. Outcome 2 is the focus of 
section 3, which is a summary of crew systems issues and future research needs that were 
derived from literature reviews, databases and aviation subject-matter experts. Finally, 
discussion and the conclusions that have been drawn are provided in section 4. 
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2. SUMMARY OF NASA STATISTICAL STUDIES 

Systems analysis personnel within the NASA Aviation Safety Program have recently conducted 
several statistical analyses of accident and incident data (ref. 3 and 4). This section summarizes 
the results of these analyses that are related to the CDM Technical Challenge. 


2.1 Analysis of NTSB/FAA Data 

In 201 1 , a study (ref. 3) was conducted to identify trends in aviation accidents and incidents that 
were related to the three VSST Technical Challenges (TCs) as they were defined at that time: 

1. Vehicle Health Assurance 

2. Effective Crew-System Interactions and Decisions in All Conditions 

3. Aircraft Loss-of-Control Prevention and Mitigation 

For the “Vehicle Health Assurance” and “Aircraft Loss-of-Control Prevention and Mitigation” 

TCs, historical aviation accidents were examined, using the National Transportation Safety 
Board (NTSB) Aviation Accident and Incident Data System (restricted to 1989-2008), and 
incidents, using the Federal Aviation Administration (FAA) Accident/Incident Data System 
(restricted to 1989-2006). The rationale was that these two TCs mapped easily to specific 
Commercial Aviation Safety Team/International Civil Aviation Organization (CAST/ICAO) (ref. 5) 
categories of accidents and incidents. The “Vehicle Health Assurance” was mapped to 
system/component failures/malfunctions (SCFM) and non-impact fires, and “Aircraft Loss-of- 
Control Prevention and Mitigation” TC to the “loss of control-inflight” category. 

However, the “Effective Crew-System Interactions and Decisions in All Conditions” TC, which is 
now the current CDM TC, was mapped to select categories of causal factors because it could 
not be associated with a particular accident category. Given that the NTSB coding process does 
not use terms such as “crew-system interaction”, the Aviation Safety Analysis and Functionality 
Evaluation (ASAFE) program (ref. 6) was used because it contains a taxonomy that maps 
specific causal factor codes used by the NTSB to more general error types based upon an 
expanded human factors analysis and classification system (HFACS) (ref. 7). For example, 
there are fourteen categories of errors specific to each of flight crew, ground personnel, air 
traffic control and maintenance. By using the ASAFE taxonomy, events with “flight crew 
decision errors” and “skill-based errors” could be selected and mapped to the “Effective Crew- 
System Interactions and Decisions in All Conditions” TC. 

In order to better understand the ASAFE taxonomy with respect to the two types of errors 
chosen for this analysis, the NTSB database was searched. This search revealed that the most 
frequently cited flight crew skill-based errors were the following: 

• failure to maintain aircraft control or directional control 

• inadequate compensation for wind conditions 

• inadvertent stall 

• inadequate preflight preparation 

• inadequate visual lookout 

In addition to the more generic errors of inadequate or poor pre-flight or in-flight planning or 
decisions, the most frequently cited specific decision errors were as follows: 

• selection of an inappropriate takeoff or landing site 

• failing to refuel the aircraft 
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• failing to perform an aborted landing or aborted takeoff 

• selection of the wrong runway 

• excessive airspeed 

• inappropriate use of certain controls (usually brakes, carburetor heat or landing gear 
extension) 

The accidents with flight crew skill-based errors and flight crew decision errors were selected if 
the error was considered either a cause or a factor (but not a finding) in the accident report. 
Causes are actions or events that are the direct source of the accident, while factors are actions 
or events that contributed to the accident (ref. 8). Each accident can have multiple causes and 
factors. Findings are actions or events that occurred in conjunction with the accident, but were 
not determined to have contributed to the accident. 

The ASAFE taxonomy cannot be applied to the incident data because the level of causal factor 
coding is far less detailed. Each incident has been assigned a general cause in one of three 
categories: individual person (pilot, maintenance crew or ground crew), operational deficiency 
(mostly non-maintenance related system component failures) or environmental (weather or 
terrain conditions). Thus we are only able to report the proportion of incidents caused by pilot 
action or inaction, without specifying the type of pilot error. 


Results 

A summary of the accidents affected by the two selected types of pilot error can be found in 
Tables 2 and 3. Denominators are included in many lines for clarification, specifically with 
respect to fatal accident percentages. There are two types of fatal accident percentages, and 
both are of interest: the percentage of all fatal accidents that included a particular type of pilot 
error, and the percentage of accidents marked by the particular pilot error in which there was a 
fatality. The numerators in these percentages are identical (fatal pilot error accidents) but the 
denominators are different. 

More than half of the accidents and fatal accidents in Part 135 and Part 91 were caused in part 
by a flight crew skill-based error (Table 2), compared with less than one quarter of Part 121 
accidents. Part 121 accidents with skill-based errors also were less likely to be fatal (7%) than 
accidents in other operations (22%-30%). 
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Table 2. Summary of Flight Crew Skill-Based Error (SBE) Accidents by Operation 

Category (1989-2008) 



Part 121 

Part 135— 
Scheduled 

Part 135— 
Non- 

Scheduled 

Part 91 

Total Flight Hours 

317,999,117 

24,446,927 

61,751,000 

507,516,000 






Total Accidents 

738 

219 

1,202 

26,922 

Accidents with SBE 

174 (23.6%) 

123 (56.2%) 

631 (52.5%) 

17962 (66.7%) 

SBE Accidents 
per million flight hrs 

0.547 

5.031 

10.218 

35.392 






Total Fatal Accidents 

67 

48 

307 

5214 

Fatal SBE Accidents 
out of All Fatal 
Accidents 

12/67(17.9%) 

29/48 (60.4%) 

190/307 

(61.9%) 

3892/5214 

(74.6%) 

Fatal SBE Accidents 
out of All SBE 
Accidents 

12/174(6.9%) 

29/123 (23.6%) 

190/631 

(30.1%) 

3892/17962 

(21.7%) 






Total Fatalities 

1956 

309 

745 

10,109 

Fatalities in accidents 
with a Skill-Based 
Error (SBE) 

122(6.2%) 

150 (48.5%) 

431 (57.9%) 

7531 (74.5%) 







Decision errors (Table 3) were noted as the cause of fewer accidents overall (13%-30%) than 
skill-based errors, but for most operation categories, a slightly higher percentage of them 
include at least one fatality (13%-29%). The differences in percentages between operational 
categories are less striking for decision errors than for skill-based errors. 
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Table 3. Summary of Flight Crew Decision Error (DE) Accidents by Operation 

Category (1989-2008) 



Part 121 

Part 135 — 
Scheduled 

Part 135 — 
Non- 

Scheduled 

Part 91 

Total Flight Hours 

317,999,117 

24,446,927 

61,751,000 

507,516,000 






Total Accidents 

738 

219 

1,202 

26,922 

Accidents with DE 

96 (13.0%) 

53 (24.2%) 

363 (30.2%) 

7306 (27.1%) 

DE Accidents 
per million flight hrs 

0.302 

2.168 

5.878 

14.395 






Total Fatal Accidents 

67 

48 

307 

5214 

Fatal DE Accidents 
out of All Fatal 
Accidents 

12/67(17.9%) 

14/48 (29.2%) 

104/307 

(33.9%) 

1694/5214 

(32.5%) 

Fatal DE Accidents 
out of All DE 
Accidents 

12/96 (12.5%) 

14/53 (26.4%) 

104/363 

(28.7%) 

1694/7306 

(23.2%) 






Total Fatalities 

1956 

309 

745 

10,109 

Fatalities in accidents 
with Decision Errors 

341 (17.4%) 

52 (16.8%) 

266 (35.7%) 

3259 (32.2%) 







Figures 2-5 display the percentage of accidents in each time period with the two types of pilot 
error as a cause or factor in the accident for each of the operational categories. In Part 121 
accidents (Figure 2), decision errors have declined substantially over time, but skill-based errors 
have consistently been a factor in twenty to twenty-five percent of accidents. 

Among Scheduled Part 135 accidents (Figure 3), the percentage of accidents with skill-based 
errors dropped during 1999-2003, but rose again in the latest time period, while decision errors 
reached a peak during 1999-2003. 

Among Non-Scheduled Part 135 accidents (Figure 4), skill-based errors have declined, and the 
percentage of accidents with decision errors has remained fairly steady. 

Among Part 91 accidents (Figure 5), there has been a decrease in the percentage of accidents 
with decision errors, but there has been little change in the percentage of accidents with skill- 
based errors. 
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1989-1993 1994-1998 1999-2003 2004-2008 

0 Skill-Based Error □ Decision Error 


Figure 2. Percentage of Part 121 Accidents with the Specified Error Type 

Across Four Time Periods. 



0 Skill-Based Error □ Decision Error 


Figure 3. Percentage of Scheduled Part 135 Accidents with the Specified Error 

Type Across Four Time Periods. 


Page 8 


Crew Decision Making Trends and Issues 



Figure 4. Percentage of Non-Scheduled Part 135 Accidents with the Specified 

Error Type Across Four Time Periods. 



s Skill-Based Error □ Decision Error 

Figure 5. Percentage of Part 91 Accidents with the Specified Error Type 

Across Four Time Periods. 

The incident data in general are not as detailed as the accident data, and that is particularly true 
with regard to causal factors. Only a single general cause is recorded for each incident, 
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although for some incidents the cause is shared (e.g., pilot and maintenance, pilot and ground 
crew). Table 4 shows the percentage of incidents for which the pilot was assigned blame, either 
solely or in part. 

The percentage of incidents caused by any pilot error ranges from 13 percent (Part 121) to 56 
percent (Part 91). In general, pilot error rates for Part 91 are roughly double those for Non- 
Scheduled Part 135, which in turn are roughly double those for Scheduled Part 135. Part 121 
incidents invoving pilot error were always less frequent than the other flight operational 
categories with one exception; errors by ground crew and pilot, which are highest in Part 121 . 
This is not surprising, since Part 121 flights are far more likely than those in other flight 
categories to require the assistance of ground crew. 


Table 4. Summary of Incidents Caused by Pilot Error by Operation Category 

(1989-2006) 


Incident Cause 

Part 121 

Part 1 35 — 
Scheduled 

Part 1 35 — 
Non- 

Scheduled 

Part 91 

Total Incidents 

7647 

1921 

2189 

27,919 

Pilot Error Only 

714 ( 9.3%) 

221 (11.5%) 

470 ( 21.5%) 

11961 (42.8%) 

Pilot Error plus Ground 
Crew 

167 ( 2.2%) 

23 ( 1.2%) 

16 ( 0.7%) 

72 ( 0.3%) 

Pilot Error plus 
Maintenance 

79 ( 1.0%) 

38 ( 2.0%) 

57 ( 2.6%) 

1142 ( 4.1%) 

Pilot Error plus Weather 

72 ( 0.9%) 

41 ( 2.1%) 

102 ( 4.7%) 

2402 ( 8.6%) 

Any Pilot Error 

1032 (13.5%) 

323 (16.8%) 

645 (29.5%) 

15577 (55.8%) 


Figures 6-9 display the percentage of incidents in each time period with the cause of the 
incident determined to be pilot error only or pilot error plus an additional factor (ground crew, 
maintenance or weather). 

Among Part 121 incidents (Figure 6), the overall percentage of incidents for which pilot error 
was the cause has decreased since 1998. It should be noted that there was a slight increase in 
the 2004-2006 time period. The percentage of incidents in which the pilot shared blame with the 
ground crew, maintenance or weather has generally increased over time. 

Among Scheduled Part 135 incidents (Figure 7), the proportion of incidents caused by pilot error 
alone has been substantially higher since 1998 than it was prior to 1998. 

Among Non-Scheduled Part 135 incidents (Figure 8), the percentage of incidents with the single 
cause of pilot error has increased consistently until the last time period of 2004-2006, just as the 
percentage of pilot plus caused incidents has decreased 
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1989-1993 1994-1998 1999-2003 2004-2006 

0 Pilot Error Only □ Pilot Error Plus 


Figure 6. Percentage of Part 121 Incidents with Pilot Error Across Four Time 

Periods (1989-2006) 


30 



1989-1993 1994-1998 1999-2003 2004-2006 


ei Pilot Error Only □ Pilot Error Plus 


Figure 7. Percentage of Scheduled Part 135 Incidents with Pilot Error 
Across Four Time Periods (1989-2006) 
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1989-1993 1994-1998 1999-2003 2004-2006 


0 Pilot Error Only □ Pilot Error Plus 


Figure 8. Percentage of Non-Scheduled Part 135 Incidents with Pilot Error 
Across Four Time Periods (1989-2006) 



0 Pilot Error Only 


□ Pilot Error Plus 


Figure 9. Percentage of Part 91 Incidents with Pilot Error Across Four Time 

Periods (1989-2006) 


Among Part 91 incidents (Figure 9), both pilot error percentages have remained relatively 
steady over time. 
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2.2 NASA Analysis of ASRS Data 

In 2011, AvSP systems analysis personnel completed a study (ref. 4) of Aviation Safety 
Reporting System (ASRS) data. The human factors related incidents were analyzed because 
the system analysis personnel believed these incidents were applicable to the “Effective Crew- 
System Interactions and Decisions in All Conditions” (currently known as CDM) TC. In 
response to the systems analysis team’s request for incident reports involving human factor 
related incidents during the time period January, 1993 through January, 2011, the analysts at 
ASRS provide a dataset of 2,243 incidents for Part 121, 135, and 91 operations. The category 
of human factors is a new category added to the search criteria for the ASRS data and is only 
available from May, 2009, instead of January, 1993. The data prior to this most likely also has 
human factor information in it, but this information would require using a free text search string 
or by reading through each report. 

Information provided included the incident report number, date of incident, phase of flight, 
aircraft make/model, SCFM, and human factors. The human factor categories used in the 
analysis are the following, in alphabetical order: 

1. Communication Breakdown 

2. Confusion 

3. Distraction 

4. Fatigue 

5. Human-Machine Interface 

6. Other/ Unknown 

7. Physiological-Other 

8. Situational Awareness 

9. Time Pressure 

10. Training / Qualification 

1 1 . Troubleshooting 

12. Workload 
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The human factors category can list more than one human factor involved for each incident 
report. A more in depth analysis of this raw data set was conducted to look at the conditions 
present when human factor incidents occurred. This information may be useful to researchers 
in determining areas of future research. 


Results 

The result of breaking the data into the human factor categories can be seen in Figure 10. In 
many of the reported incidents there were multiple human factors listed. There were 2,152 
reports that listed the specific human factor, with a total of 5,949 human factors listed in those 
2,152 reports. Situational Awareness was the tallest pole (with 1,309 incidents), followed by 
communication breakdown (844 incidents), confusion (757 incidents) human-machine interface 
(510 incidents), and distraction (506 incidents). Fatigue and physiological-other had the fewest 
reports with 95 and 84 incidents respectively. 


Human Factor Related Incidents 
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Figure 10. Human Factor Frequency for Human Factor Related Incidents 


Forty four percent of the human factors related data, or 977 incidents, have an aircraft system 
component failure listed. The remaining 1266 human factor incidents did not list an SCFM. 
Figure 1 1 shows the results for the SCFM breakdown. The SCFM tall poles are navigation at 
155, propulsion systems at 139, and monitoring and management at 121. 
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Figure 11. SCFM Frequency for Human Related Incidents 


The human factors per phase of flight were then broken down into FAR parts. Table5 displays 
the results for FAR part 121. The categories with more than 60 incidents are highlighted for 
emphasis. The largest categories were in the parked phase of flight: communication breakdown 
with 171 incidents and situational awareness with 168. Of the Part 121 incidents, the parked 
phase of flight had the most tall poles, and the situational awareness human factor had the 
largest number of tall poles. The human factors of fatigue and physiological-other had the 
fewest incident reports for all phases of flight: for the different phases of flight, the final approach 
and landing categories had the fewest incident reports for all of the human factors categories. 


Table 5. Frequency of Part 121 Human Factor per Phase of Flight for Human 

Factor Related Incidents 


Human Factor 

Phase of Flight - Part 121 

Taxi 

Take 

Off 

Initial 

Climb 

Climb 

Cruise 

Descent 

Initial 

Approach 

Final 

Approach 

Landing 

Parked 

Communication Breakdown 

76 

36 

24 

42 

72 

52 

40 

20 

19 

171 

Confusion 

67 

41 

32 

58 

50 

57 

37 

27 

21 

104 

Distraction 

43 

28 

26 

34 

59 

34 

36 

24 

12 

55 

Fatigue 

5 

3 

0 

6 

3 

2 

13 

6 

2 

24 

Human Machine Interface 

16 

43 

28 

52 

54 

39 

32 

24 

11 

26 

Other / Unknown 

8 

12 

13 

24 

44 

29 

18 

11 

16 

30 

Physiological-Other 

4 

6 

4 

9 

18 

10 

6 

3 

7 

12 

Situational Awareness 

101 

79 

47 

77 

113 

84 

74 

42 

46 

168 

Time Pressure 

12 

29 

19 

22 

39 

22 

20 

9 

10 

59 

T raining / Qualification 

23 

30 

16 

26 

45 

14 

21 

5 

6 

65 

T roubleshooting 

15 

29 

21 

42 

73 

19 

14 

7 

2 

63 

Workload 

21 

20 

29 

34 

56 

30 

29 

9 

10 

43 

Number of Incident Reports 

136 

130 

82 

148 

210 

144 

112 

70 

71 

259 
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Table 6 contains the results of the FAR part 135 human factor per phase of flight analysis. 
Confusion during initial climb was the largest reported incident with 11, followed by situational 
awareness during initial climb and cruise, both with 9 incidents. The phase of flight with the 
most tall poles was the initial climb, and the category of human factors with the most tall poles 
was situational awareness. Many categories had no incidents or only one incident for part 135 


Table 6. Frequency of Part 135 Human Factor per Phase of Flight for Human 

Factor Related Incidents 


Human Factor 

Phase of Flight - Part 135 

Taxi 

Take 

Off 

Initial 

Climb 

Climb 

Cruise 

Descent 

Initial 

Approach 

Final 

Approach 

Landing 

Parked 

Communication Breakdown 

2 

2 

7 

2 

2 

2 

1 

3 

1 

3 

Confusion 

1 

3 

11 

2 

5 

1 

2 

3 

3 

3 

Distraction 

1 

2 

5 

4 

4 

0 

1 

1 

2 

2 

Fatigue 

0 

0 

0 

0 

0 

2 

0 

0 

0 

1 

Human Machine Interface 

1 

3 

6 

3 

0 

2 

0 

0 

0 

1 

Other / Unknown 

0 

0 

0 

0 

3 

0 

0 

0 

0 

0 

Physiological-Other 

0 

0 

0 

0 

3 

0 

0 

0 

0 

0 

Situational Awareness 

3 

3 

9 

8 

9 

1 

2 

5 

5 

5 

Time Pressure 

1 

1 

4 

3 

3 

0 

1 

1 

1 

3 

T raining / Qualification 

0 

0 

4 

1 

1 

0 

0 

0 

0 

0 

T roubleshooting 

1 

2 

2 

3 

1 

0 

0 

0 

0 

2 

Workload 

1 

1 

8 

2 

4 

2 

1 

1 

2 

0 

Number of Incident Reports 

3 

7 

17 

8 

12 

2 

2 

6 

5 

7 


FAR part 91 human factors per phase of flight results are shown in Table 7. It had two tall poles 
both in situational awareness, one during cruise with 71 incidents and the other during landing 
with 61 incidents. For human factors, there was a higher concentration of tall poles in 
situational awareness. In the area of phase of flight the tall poles were more heavily 
concentrated during cruise. Fatigue, physiological-other, and other/unknown had the fewest 
human factor incidents per phase of flight for part 91 . The phase of flight for human factors with 
the fewest incidents was the parked phase of flight. 
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Table 7. Frequency of Part 91 Human Factor per Phase of Flight for Human 

Factor Related Incidents 


Human Factor 

Phase of Flight - Part 91 

Taxi 

Take 

Off 

Initial 

Climb 

Climb 

Cruise 

Descent 

Initial 

Approach 

Final 

Approach 

Landing 

Parked 

Communication Breakdown 

15 

16 

26 

12 

33 

19 

22 

17 

27 

7 

Confusion 

20 

12 

27 

13 

24 

17 

18 

20 

20 

6 

Distraction 

8 

10 

15 

12 

30 

13 

17 

13 

19 

3 

Fatigue 

0 

3 

0 

1 

0 

0 

1 

2 

2 

0 

Human Machine Interface 

6 

9 

25 

11 

28 

12 

20 

10 

19 

1 

Other / Unknown 

3 

3 

3 

4 

11 

6 

3 

3 

12 

0 

Physiological-Other 

1 

2 

1 

0 

2 

1 

1 

1 

1 

0 

Situational Awareness 

31 

33 

40 

22 

71 

28 

35 

27 

61 

9 

Time Pressure 

9 

6 

8 

4 

15 

9 

3 

6 

9 

0 

T raining / Qualification 

9 

13 

14 

10 

22 

10 

7 

12 

35 

7 

Troubleshooting 

4 

7 

7 

7 

31 

6 

6 

4 

9 

3 

Workload 

4 

3 

15 

12 

21 

16 

9 

8 

16 

0 

Number of Incident Reports 

40 

44 

57 

34 

102 

41 

48 

37 

92 

12 
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3. REVIEW OF CREW SYSTEMS ISSUES AND FUTURE RESEARCH 

NEEDS 

Subject-matter experts and researchers have conducted a number of studies that included 
recommendations for improving the safety of the air transportation system. This section contains 
a review of these safety priority lists, information databases and other documented references 
pertaining to aviation crew systems issues and future research needs. 

3.1 CAST Safety Enhancements Reserved for Future Implementation (SERFIs) 

The Commercial Aviation Safety Team (CAST) was formally established on June 23, 1998 (ref. 
9). Its mission is to provide government and industry leadership to develop and focus 
implementation of an integrated, data-driven strategy to improve commercial aviation safety. 

The members of CAST are a variety of stakeholders, including representatives from 
government, industry, pilot groups, air traffic controllers, and others. The Joint Implementation 
Monitoring Data Analysis Team (JIMDAT) is a working group with CAST. The JIMDAT monitors 
the implementation of the Safety Enhancements (SEs) in the CAST plan and suggests 
modifications and changes to CAST (ref. 10). 

In addition the current set of SEs in the CAST plan, there are number of SEs that could be 
added in the future. These SEs are referred to as “Safety Enhancements Reserved for Future 
Implementation” or SERFIs (ref. 1 1 ) for short. Table 8 contains a list of SERFIs that could 
possibly fall within the charter of the CDM TC. The SERFIs highlighted in the following table 
may not be currently funded by the NASA Aviation Safety Program. Detailed descriptions of 
each CDM related SERFI can be found immediately after Table 8. 
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Table 8. CDM Related CAST SERFIs 


SE# 

Title 

LOOSEC 

NASA 

Role 

22 

Flight Deck Equipment Upgrades - Existing Type Designs (4-5) 

FAA AIR-1 

S 

35 

Displays and Alerting Systems - Existing Designs 

AIA 


36 

Basic Airplane Design - Mode Confusion (1-2) 

NASA 

L 

61 

Aircraft/Vehicle Upgrade and Installation - Part 121/135 Cockpit 
Moving Map Display 

FAA AVR-1 

S 

62 

Aircraft/Vehicle Upgrade and Installation - Vehicle Moving Map 
Display 

FAA ARP 

S 

63 

Aircraft/Vehicle Upgrade and Installation - Non-121 Aircraft 
Moving Map Display 

FAA AVR-1 

S 

71 

Graphical Displays - Carry On 

NASA 

L 

72 

Graphical Displays - Panel Mounted - New Production 

NASA 

L 

110 

Cockpit Moving Map Phase 1 

FAA AVR-1 

S 

111 

Cockpit Moving Map Phase 1 & 2 

FAA AVR-1 

S 

112 

Cockpit Moving Map Phase 1, 2, & 3 

FAA AVR-1 

S 



LEGEND 

L 

NASA has a Leading role: NASA is the Lead Overall Organization for Safety Enhancement 
Completion (LOOSEC), or the Lead Organization for Output Completion for one or more 
SE Outputs, or is assigned a specific action in an Output 

S 

NASA has a Supporting role: NASA is shown as a contributing Resource in the SE or has 
Current Related Aviation Activities listed in the SE, but has no specific assigned actions 


NASA is not listed anywhere in the SE as a contributing organization 


SE falls within charter of CDM. Does not necessarily mean that work is currently funded 
by AvSP 
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SE022: Flight Deck Equipment Upgrades - Existing Type Designs 

This safety enhancement ensures altitude awareness and accomplishment of 
checklist items. This will be accomplished through the development of guidelines 
and procedures for flight deck smart alerting system design and supporting 
operational procedures and training based upon: 

1. The installation of equipment to provide automatic aural altitude alert call- 
outs on final approach or other such altitude alerting systems. 

2. The installation of automated or mechanical checklist devices to provide a 
positive means for checklist completion. 

3. Research and assessment of existing technology in flight deck smart- 
alerting system design. 


SE035: Displays and Alerting Systems - Existing Designs (R-D) 

This safety enhancement outlines research and development (R&D) topics 
needed to evaluate display and alerting systems in existing type designs to 
provide: 

1. Graphic speed trend information 

2. A pitch limit indication 

3. Bank angle limits to buffet 

4. Barber poles and amber bands on primary airspeed indications 

5. Detection and annunciation of conflicting attitude, airspeed and altitude 
data information 

6. Detection and removal of invalid attitude, airspeed and altitude data 
information (i.e.. from an internal fault) 

7 . Detection and removal of misleading attitude, airspeed and altitude data 
information (e.g., from an external sensor fault) to the extent feasible 

8. Information to perform effective manual recovery from unusual attitudes 
using chevrons, sky pointers, and/or permanent ground-sky horizon on all 
attitude indications 

9. Salient annunciation of autoflight mode changes and engagement status 
changes (e.g., blinking/colored/boxed mode information) 

10. Effective sideslip information and alerting of excessive sideslip (e.g., split 
trapezoid on attitude indicator) 

1 1. Clear annunciation of engine limit exceedances and significant thrust loss. 
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SE036: Basic Airplane Design - Mode Confusion (R-D) 

This safety enhancement outlines research and development (R&D) topics aimed 
at reducing fatal accidents due to mode confusion leading to loss of control. 
Information regarding observed instances of flight crew mode confusion (by 
airplane model) should be disseminated to operator training departments, 
manufacturers, and the aviation human factors research community. 

SE061: Cockpit Moving Map Display 

The purpose of this Safety Enhancement is to reduce runway incursions by 
improving pilot situational awareness using cockpit/vehicle moving map 
technology in commercial aircraft. The requirements for the moving map 
implementation are organized into four phases. Phase 1 will address 
development and installation of cockpit moving map (airport) displays with own- 
ship position enabled by GPS. Phase 2 will add display functionality for data- 
linked traffic, ground and air, utilizing ADS-B and TIS-B. Phase 3 will add 
functionality for runway occupancy advisory systems. Phase 4 will add 
functionality for data-linked taxi routes and clearance limits. Each phase will also 
address heads-up guidance display systems (HUDs). Each phase will require the 
continuing development and certification of cockpit display equipment and the 
formation of standards, guidelines and procedures for use of the equipment 


SE062: Airport Vehicle Moving Map Display 

The purpose of this Safety Enhancement is to reduce runway incursion incidents 
by improving ground vehicle operator situational awareness using airport moving 
map technology. 

SE063: General Aviation Aircraft Moving Map Display 

The purpose of this Safety Enhancement is to reduce runway incursion incidents 
by improving pilot situational awareness using cockpit moving map technology in 
general aviation (GA) aircraft. 

SE071: Turbulence Displays - Carry-on Equipment 

The purpose of this Safety Enhancement is to provide improved, real-time 
turbulence information to pilots, ground operations personnel, and forecasters for 
turbulence avoidance decisions and for input to turbulence forecasts. This will be 
accomplished through the improvement of automated airborne turbulence 
measurements; new flight deck displays of turbulence information; improved or 
new on-board look-ahead turbulence detection capabilities; and upgraded flight 
crew procedures for use of the improved information to avoid turbulence. 

SE071 assumes equipage with electronic flight bag (EFB) or other carry-on 
displays. The enhancement provides first-generation turbulence displays for 
these products via integration of up-linked information on existing carry-on 
displays. 
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SE072: Graphical Turbulence Displays - Panel Mounted - New Production 

The purpose of this Safety Enhancement is to provide improved, real-time 
turbulence information to pilots, ground operations personnel, and forecasters for 
turbulence avoidance decisions and for input to turbulence forecasts. This will be 
accomplished through the improvement of automated airborne turbulence 
measurements; new flight deck displays of turbulence information; improved or 
new on-board look-ahead turbulence detection capabilities; and upgraded flight 
crew procedures for use of the improved information to avoid turbulence. 

SE072 recommends panel-mounted graphical turbulence displays for new 
production airplanes. 


SE110: Moving Map Display with GPS Own-Ship Position 

The purpose of this Safety Enhancement is to reduce runway incursions by 
improving pilot situational awareness using cockpit/vehicle moving map 
technology in commercial aircraft. The requirements for the moving map 
implementation are organized into four phases. SE110 includes Phase 1 to 
address development and installation of cockpit moving map (airport) displays 
with own-ship position enabled by GPS. 


SE1 1 1: Moving Map Display with Own-Ship and Traffic Enabled by ADS-B/ 
TIS-B 

The purpose of this Safety Enhancement is to reduce runway incursions by 
improving pilot situational awareness using cockpit/vehicle moving map 
technology in commercial aircraft. The requirements for the moving map 
implementation are organized into four phases. SE111 builds upon SE110 by 
adding display functionality for data-linked traffic, ground and air, utilizing ADS-B 
and TIS-B to Phase 1, which addresses development and installation of cockpit 
moving map (airport) displays with own-ship position enabled by GPS 


SE112: Moving Map Display with Own-Ship, Traffic, and Runway 
Occupancy Advisory System 

The purpose of this Safety Enhancement is to reduce runway incursions by 
improving pilot situational awareness using cockpit/vehicle moving map 
technology in commercial aircraft. The requirements for the moving map 
implementation are organized into four phases. SE112 builds upon SE110 and 
SE1 11 by adding functionality for runway occupancy advisory systems to Phase 
1 (own-ship position enabled by GPS) and Phase 2 (data-linked traffic enabled by 
ADS-B/TIS-B). 
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3.2 Flight Deck Automation Issues (FDAI) Study 

The Flight Deck Automation Issues (FDAI) website (ref. 12) contains the results of a study 
conducted by principal investigators from Oregon State University, Research Integrators, Inc 
and Honeywell. The study (ref. 13) was funded by the FAA, Office of the Chief Scientific and 
Technical Advisor for Human Factors (AAR-100). The study, which was conducted over a ten 
year period (1997-2007), contained two phases: (1) identification of possible problems and 
concerns related to flight deck automation and (2) compilation of evidence related to flight deck 
automation issues. Two taxonomies of flight deck automation problems and concerns were 
developed based on the information obtained in phase one of the study. The “primary 
taxonomy” organizes the perceived problems and concerns into three major categories; reason 
for automation existence, automation design, and automation use. In the “alternative taxonomy” 
of flight deck automation problems and concerns, the 2,428 citations of 1 14 issues were divided 
into five distinct categories (Figure 12). The two largest of these categories, “Pilot Centered 
Problems and Concerns” and “Automation-Centered Problem and Concerns” can be furthered 
divided into the subcategories shown in Figures 13 and 14. Of the 19 sub-categories identified, 
“pilot/automation interface” was the most frequently cited problem/concern (13%). 


Orew-Oentereri 


Organization- 
Centered 
Problems and 


Other Problems 
and Concerns 



(2.5%) 


Automation- 



Figure 12. Percentage of Total Citations Related to the FDAI Alternative 
Taxonomy of Flight Deck Automation Problems and Concerns 
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7 . 1 % 


6 . 5 % 


6.6% 


8 . 6 % 


Figure 13. Percentage of the Total Citations Related to the FDAI Pilot-Centered 
Problems and Concerns Subcategories Taxonomy 


Air traffic control 
Standardization 
Pilot/automation interface 
Automation levels 
Complexity 
Automation authority 
Human-centered design 
Automation failure 
Functionality (capabilities and limitations) 


1 . 9 % 


2 . 7 % 


0 . 7 % 

3 . 7 % 



12 . 9 % 


Figure 14. Percentage of Total Citations Related to the FDAI Automation- 
Centered Problems and Concerns Subcategories Taxonomy 
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3.3 NTSB Most-Wanted List 

Every year the NTSB publishes a list of the most-wanted transportation safety improvements for 
various modes of transportation (e.g., highway). Although most of the research conducted in the 
NASA AvSP is directed toward commercial aircraft operations, a portion of the work in CDM 
may be applicable to the issue of “Improving General Aviation Safety”, which is currently on the 
NTSB’s Most Wanted List (ref. 14). Specific recommendations to address this Most Wanted 
issue include: 

• “Adequate education and training and screening for risky behavior are critical to 
improving general aviation safety. For example, guidance materials should include 
information on the use of Internet, satellite, and other data sources for obtaining weather 
information 

• Training materials should include elements on electronic primary flight displays, and 
pilots should have access to flight simulators that provide equipment-specific electronic 
avionics displays. 

• Knowledge tests and flight reviews should test for awareness of weather, use of 
instruments, and use of "glass" cockpits. 

• Need for a mechanism for identifying at-risk pilots and addressing risks so that both the 
pilot and passengers can safely fly.” 

In addition, the NTSB also issues safety recommendations as a result of accident investigations 
and other safety concerns that arise. Some recent open recommendations that could possibly 
be related to CDM research are listed in Table 9. 


Page 25 





Crew Decision Making Trends and Issues 


Table 9. Recent NTSB Recommendations Related to CDM Issues 


Recommendation # 

Recommendation 

A-1 1-092 

Require principal operations inspectors to review flight crew training 
programs and manuals to ensure training in tailwind landings is (1) 
provided during initial and recurrent simulator training; (2) to the extent 
possible, conducted at the maximum tailwind component certified for the 
aircraft on which pilots are being trained; and (3) conducted with an 
emphasis on the importance of landing within the touchdown zone, 
being prepared to execute a go-around, with either pilot calling for it if at 
any point landing within the touchdown zone becomes unfeasible, and 
the related benefits of using maximum flap extension in tailwind 
conditions. 

A-1 1-039 

Require that role-playing or simulator-based exercises that teach first 
officers to assertively voice their concerns and that teach captains to 
develop a leadership style that supports first officer assertiveness be 
included as part of the already required crew resource management 
training for 14 Code of Federal Regulations Part 121, 135, and 91 
subpart K pilots. 

A-1 1-028 

Actively pursue with aircraft and avionics manufacturers the 
development of technology to reduce or prevent runway excursions and, 
once it becomes available, require that the technology be installed. 

A-1 0-037 

Require all manufacturers of certified electronic primary flight displays to 
include information in their approved aircraft flight manual and pilot’s 
operating handbook supplements regarding abnormal equipment 
operation or malfunction due to subsystem and input malfunctions, 
including but not limited to pitot and/or static system blockages, 
magnetic sensor malfunctions, and attitude-heading reference system 
alignment failures. 
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4. DISCUSSION AND CONCLUSIONS 


The CDM TC, does not address a specific CAST/ICAO occurrence category, but instead 
addresses specific causal factors which lead to accidents: flight crew skill-based errors and 
flight crew decision errors. Skill-based errors were a causal factor in more than half of all Part 
135 accidents, two thirds of Part 91 accidents, and almost one quarter of Part 121 accidents. 

For all operational categories, accidents involving skill-based errors have not decreased 
appreciably during the study time period. Accidents involving decision errors have decreased 
for all operational categories except in Scheduled Part 135 operations. Even though more 
accidents are caused by skill-based errors, those accidents involving decision errors are more 
likely to be fatal than those accidents with skill-based errors. In Part 121 accidents, decision 
error resulted in more fatalities than those accidents with skill-based errors. 

The human factor related incidents analysis of ASRS dataset resulted in 2,243 reports. It is 
important to note that the category of human factors in the ASRS data set has only been added 
since May, 2009 and that previous data within ASRS probably has human factor related 
incidents in it but that they are not categorized as such. A larger time frame would be helpful to 
better understand the causes of human error in the incident data. The data was analyzed in 
four ways: FAR part (121, 135, and 91), 10 phases of flight, 18 SCFM categories, and 12 
human factors. There were 2,243 phase of flight incident reports, 1,966 FAR part 121, 135, and 
91 incident reports, 977 SCFM listings, and 2,152 with specific human factors categories. When 
analyzing the human factors, FAR parts, and phase of flight, situational awareness was the top 
problem for all the FAR part categories. Communication breakdown and confusion also had 
higher frequencies than the other human factor categories. There was no one phase of flight 
that had the majority of problems. The fewest human factors were seen in fatigue and 
physiological-other. Analyzing human factors and SCFMs per FAR part also showed situational 
awareness as being the largest incident issue. In the future more analysis needs to be done in 
the area of situational awareness to determine what the biggest issues are in that human factors 
category. 

A review of crew systems related issues in safety priority lists, information databases and other 
document references identified a number of areas related to the CDM TC. Out of the 47 SERFIs 
that have been identified by the CAST, 1 1 of these Safety Enhancements could possibly fall 
within the charter of the CDM TC. In addition, many of the crew related issues identified by in 
the FDAI study and in published literature are closely related to the CDM goals and products 
(Figure 1). For example, the top three automation-center problems identified in the FDAI study, 
“pilot/automation interface”, “functionality (capabilities and limitations)” and “automation failure” 
are all being addressed by researchers within CDM. Although the current CDM products 
primarily focuses on crew related technologies and capabilities, modifications can be made in 
the future so that more of the CDM portfolio addresses the NTSB’s general aviation 
recommendations. 

One of the upcoming tasks of the AvSP systems analysis personnel is to develop a Bayesian 
model similar to the one created for loss of control (ref. 15, 16, 1 7), that examines the role of 
increasing complexity and reliance on automation in the future National Airspace System. This 
model will be used by AvSP management for identifying any gaps in its research portfolio and 
also to evaluate the impact of its research products on reducing the risk of an aviation accident 
in the future. The FDAI “alternative taxonomy” was selected as a foundation for organizing 
automation related issues in the “complexity and reliance on automation” Bayesian model 
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currently under development. However, the FDAI “alternative taxonomy” was lacking some 
characteristics needed for the modeling the effort such as time-dependencies (e.g., latent vs. 
active causes), a further breakdown of automation-centered issues, etc. Therefore, the AvSP 
systems analysis personnel combined the FDAI “alternative taxonomy” with other related 
research pertaining to human factors, automation and flight deck design (ref. 18, 19, 20), to 
create the list of automation issues outlined in Table 10. 


Table 10. Summary of Automation Issues 


Automation 
Centered Issues 

Operations 

Pilot Centered 
Issues 

Organization- 

Centered 

Issues 

Automated 
Aircraft/ ATC 
Interaction- 
Centered 
Issues 

Flight Crew 

Interface/Design 

■ Development 
process 

■ Pilot/automation 
interface 

■ Human- 
centered design 

General 

Automation 

Design 

■ Functionality 

■ Complexity 

■ Levels of 
automation 

■ Automation 
failure 

Underlying 

Philosophy of 

Automation 

■ Standardization 

■ Pilot/automation 
responsibility 
and authority 

■ Pilot role 

Crew 

Performance 

■ Crew 
coordination 

■ Performance 

■ Use of 
automation 

Awareness 

■ Situation/energy 
awareness 

■ Automation 
awareness 

Workload 

* Attention 

* Workload 

■ Skill 

■ Confidence/trust 

■ Understanding 
of automation 

■ Crew personnel 
factors 

■ Company 
automation 
philosophies, 
policies , and 
procedures 

■ Pilot selection, 
training, and 
evaluation 

* ATC conflicts 
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